Method and apparatus for unsupervised training of natural language processing units

ABSTRACT

A method of training a natural language processing unit applies a candidate learning set to at least one component of the natural language unit. The natural language unit is then used to generate a meaning set from a first corpus. A second meaning set is generated from a second corpus using a second natural language unit and the two meaning sets are compared to each other to form a score for the candidate learning set. This score is used to determine whether to modify the natural language unit based on the candidate learning set.

REFERENCE TO RELATED APPLICATIONS

The present application is a divisional of and claims priority from U.S.patent application Ser. No. 09/849,833, filed on May 4, 2001 andentitled “Method and Apparatus for Unsupervised Training of NaturalLanguage Processing Units”.

BACKGROUND OF THE INVENTION

Natural language understanding involves converting a string ofcharacters into a meaning set representing the meaning of the string ofcharacters. Such processing can involve a number of natural languagecomponents including a segmentation component that assigns characters toindividual words, a part of speech tagger that identifies the part ofspeech of each word, a syntactic parser that assigns a structure to asentence or group of sentences so that the syntactic relationshipbetween the words can be understood and a semantic interpreter thatanalyzes the syntactic parse to produce a semantic structure.

Each component in a natural language system must be trained before itcan be used. In the past, such training has largely been done by hand.For example, the rules used by syntactic parsers to parse sentences werederived by hand. However, training by hand is a laborious process oftrial and error. Because of this, more recent systems have attempted todevelop natural language components automatically, using supervisedmachine learning techniques for training.

For example, in supervised training of a parser, a corpus of inputsentences is created that is annotated to indicate the syntacticstructure of each sentence. Such annotated sentences are referred to astree banks in the art. During training, proposed changes to the parsingrules, known as candidate learning sets, are tested by repeatedlyparsing the tree bank using a different candidate learning set for eachparse. The candidate learning set that provides the best parse based onthe annotations in the tree bank is then used to change the parserrules.

One problem with using supervised training is that it is expensive andtime-consuming. For example, tree banks are so expensive andtime-consuming to create that there are very few in existence in theworld.

Thus, a less expensive and less time-consuming method is needed fortraining natural language processing components.

SUMMARY OF THE INVENTION

A method of training a natural language processing unit applies acandidate learning set to at least one component of the natural languageunit. The natural language unit is then used to generate a meaning setfrom a first corpus. A second meaning set is generated from a secondcorpus using a second natural language unit and the two meaning sets arecompared to each other to form a score for the candidate learning set.This score is used to determine whether to modify the natural languageunit based on the candidate learning set.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a general computing environment in whichembodiments of the present invention may be practiced.

FIG. 2 is a flow diagram of a method of training a natural language unitunder one embodiment of the present invention.

FIG. 3 is a block diagram of a training system under one embodiment ofthe present invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

FIG. 1 illustrates an example of a suitable computing system environment100 on which the invention may be implemented. The computing systemenvironment 100 is only one example of a suitable computing environmentand is not intended to suggest any limitation as to the scope of use orfunctionality of the invention. Neither should the computing environment100 be interpreted as having any dependency or requirement relating toany one or combination of components illustrated in the exemplaryoperating environment 100.

The invention is operational with numerous other general purpose orspecial purpose computing system environments or configurations.Examples of well known computing systems, environments, and/orconfigurations that may be suitable for use with the invention include,but are not limited to, personal computers, server computers, hand-heldor laptop devices, multiprocessor systems, microprocessor-based systems,set top boxes, programmable consumer electronics, network PCs,minicomputers, mainframe computers, telephony systems, distributedcomputing environments that include any of the above systems or devices,and the like.

The invention may be described in the general context ofcomputer-executable instructions, such as program modules, beingexecuted by a computer. Generally, program modules include routines,programs, objects, components, data structures, etc. that performparticular tasks or implement particular abstract data types. Theinvention may also be practiced in distributed computing environmentswhere tasks are performed by remote processing devices that are linkedthrough a communications network. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including memory storage devices.

With reference to FIG. 1, an exemplary system for implementing theinvention includes a general purpose computing device in the form of acomputer 110. Components of computer 110 may include, but are notlimited to, a processing unit 120, a system memory 130, and a system bus121 that couples various system components including the system memoryto the processing unit 120. The system bus 121 may be any of severaltypes of bus structures including a memory bus or memory controller, aperipheral bus, and a local bus using any of a variety of busarchitectures. By way of example, and not limitation, such architecturesinclude Industry Standard Architecture (ISA) bus, Micro ChannelArchitecture (MCA) bus, Enhanced ISA (EISA) bus, Video ElectronicsStandards Association (VESA) local bus, and Peripheral ComponentInterconnect (PCI) bus also known as Mezzanine bus.

Computer 110 typically includes a variety of computer readable media.Computer readable media can be any available media that can be accessedby computer 110 and includes both volatile and nonvolatile media,removable and non-removable media. By way of example, and notlimitation, computer readable media may comprise computer storage mediaand communication media. Computer storage media includes both volatileand nonvolatile, removable and non-removable media implemented in anymethod or technology for storage of information such as computerreadable instructions, data structures, program modules or other data.Computer storage media includes, but is not limited to, RAM, ROM,EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical disk storage, magnetic cassettes,magnetic tape, magnetic disk storage or other magnetic storage devices,or any other medium which can be used to store the desired informationand which can be accessed by computer 110. Communication media typicallyembodies computer readable instructions, data structures, programmodules or other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. The term “modulated data signal” means a signal that has one ormore of its characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, RF,infrared and other wireless media. Combinations of any of the aboveshould also be included within the scope of computer readable media.

The system memory 130 includes computer storage media in the form ofvolatile and/or nonvolatile memory such as read only memory (ROM) 131and random access memory (RAM) 132. A basic input/output system 133(BIOS), containing the basic routines that help to transfer informationbetween elements within computer 110, such as during start-up, istypically stored in ROM 131. RAM 132 typically contains data and/orprogram modules that are immediately accessible to and/or presentlybeing operated on by processing unit 120. By way o example, and notlimitation, FIG. 1 illustrates operating system 134, applicationprograms 135, other program modules 136, and program data 137.

The computer 110 may also include other removable/non-removablevolatile/nonvolatile computer storage media. By way of example only,FIG. 1 illustrates a hard disk drive 141 that reads from or writes tonon-removable, nonvolatile magnetic media, a magnetic disk drive 151that reads from or writes to a removable, nonvolatile magnetic disk 152,and an optical disk drive 155 that reads from or writes to a removable,nonvolatile optical disk 156 such as a CD ROM or other optical media.Other removable/non-removable, volatile/nonvolatile computer storagemedia that can be used in the exemplary operating environment include,but are not limited to, magnetic tape cassettes, flash memory cards,digital versatile disks, digital video tape, solid state RAM, solidstate ROM, and the like. The hard disk drive 141 is typically connectedto the system bus 121 through a non-removable memory interface such asinterface 140, and magnetic disk drive 151 and optical disk drive 155are typically connected to the system bus 121 by a removable memoryinterface, such as interface 150.

The drives and their associated computer storage media discussed aboveand illustrated in FIG. 1, provide storage of computer readableinstructions, data structures, program modules and other data for thecomputer 110. In FIG. 1, for example, hard disk drive 141 is illustratedas storing operating system 144, application programs 145, other programmodules 146, and program data 147. Note that these components can eitherbe the same as or different from operating system 134, applicationprograms 135, other program modules 136, and program data 137. Operatingsystem 144, application programs 145, other program modules 146, andprogram data 147 are given different numbers here to illustrate that, ata minimum, they are different copies.

A user may enter commands and information into the computer 110 throughinput devices such as a keyboard 162, a microphone 163, and a pointingdevice 161, such as a mouse, trackball or touch pad. Other input devices(not shown) may include a joystick, game pad, satellite dish, scanner,or the like. These and other input devices are often connected to theprocessing unit 120 through a user input interface 160 that is coupledto the system bus, but may be connected by other interface and busstructures, such as a parallel port, game port or a universal serial bus(USB). A monitor 191 or other type of display device is also connectedto the system bus 121 via an interface, such as a video interface 190.In addition to the monitor, computers may also include other peripheraloutput devices such as speakers 197 and printer 196, which may beconnected through an output peripheral interface 190.

The computer 110 may operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer180. The remote computer 180 may be a personal computer, a hand-helddevice, a server, a router, a network PC, a peer device or other commonnetwork node, and typically includes many or all of the elementsdescribed above relative to the computer 110. The logical connectionsdepicted in FIG. 1 include a local area network (LAN) 171 and a widearea network (WAN) 173, but may also include other networks. Suchnetworking environments are commonplace in offices, enterprise-widecomputer networks, intranets and the Internet.

When used in a LAN networking environment, the computer 110 is connectedto the LAN 171 through a network interface or adapter 170. When used ina WAN networking environment, the computer 110 typically includes amodem 172 or other means for establishing communications over the WAN173, such as the Internet. The modem 172, which may be internal orexternal, may be connected to the system bus 121 via the user inputinterface 160, or other appropriate mechanism. In a networkedenvironment, program modules depicted relative to the computer 110, orportions thereof, may be stored in the remote memory storage device. Byway of example, and not limitation, FIG. 1 illustrates remoteapplication programs 185 as residing on remote computer 180. It will beappreciated that the network connections shown are exemplary and othermeans of establishing a communications link between the computers may beused.

The present invention provides a method and apparatus for performingunsupervised training of one or more natural language processingcomponents, such as syntactic parsers and/or semantic interpreters. Theinvention performs this training by utilizing at least two naturallanguage processing systems, typically consisting of a syntactic parserand semantic interpreter, possibly with other components. These systemsare used to form separate meaning sets from parallel corpora, whichrepresent the same set of sentences written in different languages.Thus, one natural language processing system generates a meaning setfrom the corpus written in a first language, such as English, while asecond natural language processing system generates a meaning set fromthe corpus written in a second language, such as French. The two meaningsets are compared and the configuration of one or more of the componentsof the natural language processing systems is adjusted so that themeaning sets converge toward each other. The configurations for thecomponents that provide the best convergence for the meaning sets arethen selected and the training ends.

The method and apparatus of the present invention are described belowwith reference to the flow diagram of FIG. 2 and the block diagram ofFIG. 3.

In the block diagram of FIG. 3, two sets of natural language processingunits 300 and 302 are provided. Each natural language processing unitderives a meaning set from a corpus written in a separate language.Thus, natural language processing unit 300 derives a meaning set 304from a corpus 306 written in a language “S”, while natural languageprocessing unit 302 derives a meaning set 308 from a corpus 310 writtenin a language “E”. Note that language “S” and language “E” can be anyknown written language.

Language S corpus 306 and language E corpus 308 contain sentencesexpressing identical meaning written in different languages under mostembodiments. Thus, together they form a single bilingual corpus. Thesentences in each corpus are aligned such that a sentence or groups ofsentences that convey a meaning in one corpus are aligned with thesentences or groups of sentences that convey the same meaning in theother corpus. For instance, if the meaning in the first sentence incorpus 306 is the same as the meaning in the first sentence in corpus308, the first sentence in corpus 306 is aligned with the first sentencein corpus 308. Note that if the meaning found in one sentence of acorpus is expressed in two sentences in the other corpus, the singlesentence of the first corpus would be aligned with the two sentences ofthe second corpus.

Natural language processing unit 300 includes a syntactic parser 320 anda semantic interpreter 322 for language “S”. The operation of syntacticparser 320 and semantic interpreter 322 are controlled by a parserspecification 324 and an interpreter specification set 326,respectively, where each specification defines the input/output mappingsof the component. For example, the interpreter specification defines themeaning sets that will be produced at the output of the interpreter forvarious input semantic structures.

Similarly, natural language processing unit 302 includes a language “E”syntactic parser 330 and a semantic interpreter 332 that are controlledby a parser specification 334 and an interpreter specification 336,respectively.

Note that natural language processing units 300 and 302 can also includeother natural language components such as a part of speech tagger or asegmentation component. Although these additional components are notshown in FIG. 3 for simplicity, those skilled in the art will recognizethat the present invention can be applied to any of the trainablenatural language components that are present in a natural languageunderstanding unit.

Under the method of the present invention, one or more of thespecifications 324, 326, 334 and/or 336 are adjusted throughunsupervised training. In the description below, an unsupervisedtraining method involving generating and testing candidate learning setsis described. However, those skilled in the art will recognize that thepresent invention may be incorporated in other unsupervised trainingtechniques such as greedy hill climbing and variants of theexpectation-maximization algorithm.

The generating and testing embodiment of the present invention is shownin FIG. 2 and begins at step 200 where an unsupervised learning module352 in FIG. 3 selects an initial specification for each of the parsersand interpreters. In addition, learning module 352 selects one or morelearning sets, which are candidate changes that are to be tested usingthe process of FIG. 2. Specifically, each learning set will beindividually applied to the various natural language componentspecifications. By applying each learning set, the training method ofthe present invention is able to determine which candidate changeprovides the best improvement in the natural language units.

Using the selected specifications, the process of FIG. 2 continues atstep 201 where natural language processing unit 300 performs naturallanguage processing on language “S” corpus 306 to produce a baselinemeaning set 304. At step 202, natural language processing unit 302performs natural language processing on language “E” corpus 308 toproduce a baseline meaning set 310.

Once each natural language processing unit has formed its meaning sets,the meaning sets are compared to one another at step 204 by a scorecalculator 350 to generate a combined score for the two meaning sets.Under one embodiment, this score is generated using the followingdistance calculation: $\begin{matrix}{{{MR}\quad{Distance}} = \frac{\sum\limits_{i = 1}^{N}{{Sim}\left( {M_{i}^{E},M_{i}^{S}} \right)}}{N}} & {{EQ}.\quad 1}\end{matrix}$where Sim(M_(i) ^(E), M_(i) ^(S)) is a similarity function that providessome measure of the similarity between a meaning M_(i) ^(E) for asentence “i” in language corpus E and a meaning M_(i) ^(S) forcorresponding sentence or sentences i in language corpus S. In Equation1, the sum is performed over all N sentences in each corpus, with thedivision providing an average similarity or distance score.

After the score for the current component specifications have beendetermined, unsupervised learning module 352 determines whether there isa learning set that should be tested at step 206. If there is a learningset that needs to be tested, the process continues at step 208 wherelearning module 252 changes one or more of the component specificationsto implement the learning set to be tested. The process then repeatssteps 201, 202 and 204 by processing the bilingual language corpus usingthe current learning set and comparing the resulting meaning sets toproduce a score for the current learning set.

Steps 201, 202, 204, 206 and 208 are repeated until a score has beengenerated for each candidate learning set. When there are no morelearning sets to be tested at step 206, the process continues at step210 where the learning set that provides the highest score is selected.The appropriate component specifications are then changed to implementthe selected learning set.

The process of FIG. 2 can be repeated for multiple learning sets,thereby progressively improving the component specifications of thenatural language units.

Note that although two natural language units were used in FIGS. 2 and3, in other embodiments, larger numbers of natural language units areused. In such embodiments, the learning sets can be limited to a singlenatural language unit or may be distributed across several naturallanguage units.

In embodiments that limit the learning sets to a single natural languageunit, a separate distance measure is determined between the changingnatural language unit and all of the other natural language units. Underone embodiment, these separate distance scores are averaged to form asingle distance score for the current learning set. In otherembodiments, the maximum or minimum distance score is selected as thedistance score for the learning set.

Because the present invention adjusts each natural language componentspecification based on the meaning set provided by the natural languageunit, each component is improved so that it provides a better meaningset instead of just a better output from the component itself. Sinceproducing a good meaning set is goal of a natural language understandingunit, it is thought that improving each component based on the outputmeaning set produces more useful components for natural languageprocessing units.

Note that once a natural language component has been trained under thepresent invention, it may be used outside of natural languageunderstanding. Thus, a syntactic parser trained under the presentinvention may be used in a grammar checker that does not include asemantic interpreter.

Although the invention above was described in terms of generating ameaning set, in other embodiments, the natural language unit generates aset of actions based on the language corpus instead of a meaning set.For example, in an embodiment where the natural language unit is used tocontrol a robot, the output of the natural language unit is a set ofactions performed by the robot. In such embodiments, the set of actionsproduced by natural language units 300 and 302 are compared to eachother to generate a score instead of using a meaning set. This score isthen used to modify one or more of the natural language units so thatthe actions produced by the two units are more similar.

Although the present invention has been described with reference toparticular embodiments, workers skilled in the art will recognize thatchanges may be made in form and detail without departing from the spiritand scope of the invention.

1. A method of training a natural language unit comprising: generating afirst meaning set from a first corpus using a first natural languageunit; generating a second meaning set from a second corpus using asecond natural language unit; comparing the first meaning set to thesecond meaning set to generate a score; and using the score to determinehow to modify the first natural language unit.
 2. The method of claim 1wherein the first corpus comprises a corpus written in a first languageand the second corpus comprise the corpus written in a second language.3. The method of claim 2 wherein the second corpus is aligned with thefirst corpus.
 4. A computer-readable medium having computer-executableinstructions for performing steps for training natural language units,the steps comprising: converting a corpus of sentences into at least twomeaning sets using at least two different natural language units; andcomparing the meaning sets to evaluate the performance of one or more ofthe at least two natural language units.
 5. The computer-readable mediumof claim 4 wherein converting a corpus of sentences comprises convertinga corpus comprising sentences from at least two different languages. 6.A method of training a natural language unit comprising: generating afirst action set from a first corpus using a first natural languageunit; generating a second action set from a second corpus using a secondnatural language unit; comparing the first action set to the secondaction set to generate a score; and using the score to determine how tomodify the first natural language unit.
 7. The method of claim 6 whereinthe first corpus comprises a corpus written in a first language and thesecond corpus comprise the corpus written in a second language.